In the Specification 
Kindly replace paragraphs [0001] through [0022] with the following: 
Related Application 

This is a $371 of International Application No. PCT/FR2005/000428, with an international 
filing date of February 23, 2005 (WO 2005/091 104 A2. published September 29, 2005), which is 
based on French Patent Application No. 04/50329, filed February 23, 2004. 
Technical Field 

The pfesentinvention relates to th e ar e a of musical controllers . Th e pr e s e nt inv e ntion relat e s 
more ^ particularly to an man machin e interface permitting, e.g., the control of music software or of a 
controller by a multi-contact tactile screen with the manipulation of virtual objects. 
Background 

Th e prior art alr e ady contains manual Manual -type software controllers are known . They 
ar einclude , e.g., potentiometers that can be manipulated by the user in the form of a console and 
control the different functions of music software. Such a console forms, e .g., th e subject matt e r of 
PCT application is disclosed in WO 01/69399. 

TheOne disadvantage of this type of controller is that they are not very ergonomic for an 
efficient manipulation of software. Th e solution propos e d by th e pr e s e nt inv e ntion is One thought 
has been to implement a tactile screen for the manipulation of and the access to software functions. 

In the area of tactile controllers the prior art alr e ady contains in PCT application^ WO 
03/041006 erand US 6,570,078 disclose musical controllers with tactile control on a matrix sensor. 
The technologies described there in these docum e nts permit [[a ]]tactile control of the multi-contact 
type in which all the fingers can intervene for the control of software. 
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However, th e s e dooumonts those publications do not propos e contemplate a visual return of 
the manipulations since the different matrix sensors are of the opaque type. 

The prior art contains in Am e rican patent application US 2002/005108 (Lest e r Franck 
Ludwig) "Tactile, Visual and Array Controllers for Real Time Control of Music Signal Processing, 
Mixing, Video and Lighting " discloses a system and a process for controlling in real time signal 
processors, synthesizers, musical instruments, MIDI processors, lights, video, and special effects 
during presentations, recordings or in compositional environments using images derived from tactile 
sensors, from matrices of pressure sensors, from matrices of optical transducers, from matrices of 
chemical sensors, matrices of body sensors and from digital processes. Th e invention of this 
Am e rican pat e nt application That system furnishes touchpads, matrices of pressure sensors and 
matrices of body sensors as interfaces of tactile control, video cameras and matrices of light sensors 
such as optical transducers, matrices of chemical sensors and of other apparatuses for generating 
digital images from processes on computers or from digital simulations. The tactile transducers can 
be arranged on the keys of conventional instruments, fee-attached to existing instruments or also be 
used to create new instruments or new controllers. The matrices of chemical sensors and the other 
apparatuses for generating digital images from computer processes or from digital simulations can be 
used to observe or simulate natural physical phenomena such as environmental conditions or self- 
organizing process behaviors. Matric e s of scalars Scalar matrices or e£vectors are processed in ord e r 
to extract pattern limits, geometric properties of pixels within limits (geometric center, weighted 
moments, etc.) and information derived from a higher level (direction of rotation, segmented regions, 
pattern classification, syntax, grammars, sequences, etc.) that are used to create control signals to 
external video and visual equipment and for control or even algorithms. This inv e ntionl t also 
aHews provides MIDI and non-MIDI control signals to b e furnished . 
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This American pat e nt application^ does not pfepese contemplate a visual return of 
manipulations . This American pat e nt application and does not mention a command law. Finally, 
this pat e nt application ^ does not propo se contemplate technical solutionis]] to the masking 
phenomena that intervene when several figures are aligned or placed in an orthogonal manner on the 
sensor. The resolution of these problems is indispensable for realizing a multi-contact tactile sensor. 

Th e prior art also contains in Am e rican patent US 5,027,689 (Yamaha) "Musical Tone 
Generating Apparatus " discloses an apparatus for generating musical sounds. ThisThat apparatus 
comprises a device for generating positional information for generating information about the 
position of musical instruments (PS) as values of plane coordinates. This information (PS) is stored 
in a memory device or determined in a selective manner by a manual operation. The apparatus also 
comprises a device for the conversion of information for converting the information (PS) into 
information for controlling parameters of musical sounds (PD). This PD control information 
controls the source signals of musical sounds (SI 1, S12 and SI 3) for generating a sound field 
corresponding to the position of musical instruments arranged on a stage. This allows an operator to 
verify the positions of musical instruments on a stage, thus supplying the sensation of being in a true 
live performance. 

This Am e rican pat e nt lt mentions a multi-contact^ but it is only two contacts on an axis and 
not in Cartesian coordinates. The apparatus of this Am e rican pat e nt only functions linearly for the 
multipoint option and does not allow tracking (following of trajectory). Moreover, the apparatus ef 
this Am e rican pati e nt requires a plurality of sensors specific to each of the instruments wh e r e as the 
pr e s e nt inv e ntion r e lat e s to a g e n e ric s e nsor . 

Th e prior art or also contains US 5,559,301 discloses a solution of the musical controller type 
in the form of a tactile screen with visual return of the manipulated objects by US patent 5, 559, 301 . 



5 



However, this pat e nt it describes predefined objects (essentially of the sliders type and circular 
potentiometer type). These object types are rathe^limiting and can prove to be not very ergonomic 
for special manipulations. Moreover, the acquisition mode described in this pat e nt is not in real 
time. In fact, an icon must first be activated by a first contact with a finger, then the manipulated 
object, and the values are only updated after the icon has been released. This solution That system 
does not allow [[a ]]management in real time of the parameters associated with the object. Finally, 
the tactile sensor us e d in this patent is a "mono-contact' 9 sensor that permits the acquisition, e.g., 
only for a single finger and therefore the control of a single object at a time. This characteristic is 
very limiting for an efficient manipulation of objects. 
Summary 

This invention relates to a process for controlling computerized equipment with a device 
including a multi-contact bidimensional sensor that acquires tactile information and a calculator that 
generates command signals as a function of the tactile information, including generating graphical 
objects on a screen placed under a transparent multi-contact tactile sensor, each graphical object 
associated with at least one specific processing rule such that the sensor delivers during each 
acquisition phase a plurality of tactile information, and each piece of the tactile information forms an 
object of a specific processing determined by its localization relative to a position of one of the 
graphical objects. 

This invention also relates to a device for controlling computerized equipment including a 
multi-contact bidimensional sensor for acquisition of tactile information, a viewing screen arranged 
under the bidimensional tactile sensor, a memory for recording graphical objects that are each 
associated with at least one processing rule, and a local calculator that analyzes positions of acquired 
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tactile information and applies a processing rule as a function of the position relative to the position 

of the graphical objects. 

Brief Description of the Drawings 

The disclosure will be better understood with the aid of the following description given 
below solely by way of explanation of a selected, representative example with reference made to the 
attached figures in which: 

Fig. 1 A is a functional diagram of a controller; 

Fig. IB represents the structure of the controller associated with the functional diagram; 

Fig. 1C represents the functional diagram of the different stages of the processes for the 
acquisition of data coming from the sensor, of the creation of cursors associated with the different 
fingers, of the interaction with the graphical objects and of the generation of control messages; 

Fig. 2A is a description of the tactile matrix sensor; 

Fig. 2B describes the first stage of the scanning functioning of the sensor in order to obtain 
the multi-contact information; 

Figs. 2C, 2E and 2F explain the resolution of problems of orthogonality; 
Fig. 2D is a functional diagram of the capture interface; 

Figs. 3 A to 3F are diagrams explaining the stages for the creation of cursors, filtering, 
calculation of barycenter, mapping and of the control of graphical objects; 
Figs. 4 and 5 represent different examples of graphical objects; 

Figs. 6 to 10 represent different examples of combinations of graphical objects on the 
controller; and 

Fig. 1 1 illustrates the network use of the controller associated with the computer of the user. 
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Detailed Description 

In all th e following the The term "multi-contact" defines a tactile sensor that allows the 
acquisition of contact zones of several fingers at a time in contrast to "mono-contact" sensors that 
only allow the-acquisition for a single finger or for a stylus as, e.g., in the pr e ceding pat e nt US 5, 
559, 301. 

Th e pres e nt inv e ntion has the problem of rectifying the disadvantages of the prior art by 
proposin g We provide a screen for multi-contact tactile musical control with visual return of the 
different actions of the user on parameterable objects. 

In ord e r to do this th e pr e s e nt inv e ntion is of th e typ e d e scrib e d abov e and is remarkabl e in its 
broad e st m e aning in that it conc e rns We also provide a process for the control o f controlling 
computerized equipment b ywith a device comprising a multi-contact bidimensional sensor for the 
acquisition of tactile information as well as compris e s calculating means generating command 
signals as a function of this tactile information, characterized in that it compris e s and a stage for the 
g e n e ration o f generating graphical objects on a screen placed under a transparent multi-contact tactile 
sensor, each of which graphical objects is associated with at least one specific processing law, 
that wherein the sensor delivers during each acquisition phase a plurality of tactile information, and 
thateach piece of thisthe tactile information forms the object of a specific processing determined by 
its localization relative to the position of one of these graphical objects. 

The processings proferablv process steps may comprise a bounding zone detection of the 
contact zone of an object with the tactile sensor. 

The proc e ssings advantageouslv process may also comprise a barycenter detection. 

It preforabl v mav further comprise[[s]] stages for the-refreshing ^graphical objects as a 
function of the processings carried out during at least one previous acquisition stage. 



8 



According to an embodim e nt it The process may comprise[[s]] a stage for editing graphical 
objects consisting in cluding generating a graphical representation from a library of graphical 
components and functions and ift-determining an associated processing law. 

The acquisition frequency of the tactile information is preferabl v mav be greater than 50 Hz. 

Th e pr e s e nt inv e ntion We also conc e rns provide a device for th e control o f controlling a 
computerized piece of equipment comprising a multi-contact bidimensional sensor for the 
acquisition o f acquiring tactile information, charact e riz e d in that it furth e rmor e compris e s a viewing 
screen arranged under the bidimensional tactile sensor, as w e ll as a memory for recording graphical 
objects that are each associated with at least one processing law, and a local calculator for analyzing 
the position of acquired tactile information and the-application of a processing law as a function of 
thisthe position relative to the position of the graphical objects. 

Mor e ov e r, it is pr e f e rabl v The device may be connected to a hub (multi-socket network) fef 
formin g to form a network of controllers. 

Kindly replace paragraphs [0024] through [0027] with the following: 

Furthermore, thisthe device pr e f e rabl v mav comprise[[s]] a network output suitable for 
receiving a network cable. 

Th e inv e ntion will b e b e tter und e rstood with the aid of th e following d e scription given b e low 
sol e ly by way of e xplanation of an e mbodim e nt of th e inv e ntion with r e fer e nce mad e to the attach e d 
figures in which: 

Figur e 1 A is a functional diagram of th e controll e r in accordanc e with th e invention. 
Figur e IB repr e s e nts th e structure of th e controll e r associated with th e functional diagram of 
th e invention. 
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Figur e 1C repr e s e nts the functional diagram of the differ e nt stages of tho processes for th e 
acquisition of data coming from th e s e nsor, of th e cr e ation of cursors associated with th e diff e r e nt 
fingers, of th e interaction with tho graphical obj e cts and of th e gen e ration of control m e ssag e s. 

Figur e 2A is a d e scription of th e tactil e matrix s e nsor. 

Figur e 2B d e scrib e s th e first stag e of th e scanning functioning of the s e nsor in ord e r to obtain 
th e multi contact information. 

Figur e s 2C, 2E and 2F e xplain the r e solution of problems of orthogonality. 
Figur e 2D i s a functional diagram of th e captur e int e rfac e . 

Th e series of figur e s 3A to 3F e xplain th e stag e s for th e cr e ation of cursors, filt e ring, 
calculation of baryc e nt e r, mapping and of th e control of graphical obj e cts. 
Figur e s 4 , 5 repr e s e nt differ e nt e xampl e s of graphical obj e cts. 

Figur e s 6 to 10 r e pr e sent diff e r e nt e xamples of combinations of graphical obj e cts on th e 
controll e r. 

Figur e 1 1 illustrat e s the n e twork us e of th e controll e r associat e d with th e comput e r of th e 

In all of the following description, the control is performed on a computerized piece of 
equipment that can be, e.g., a music software, a controller, audiovisual equipment or multimedia 
equipment. 

As figuresFigs. 1 A, IB and more pr e cisely 2A illustrate, the first basic element of th e pr e sent 
inv e ntion is the matrix sensor 101 necessary for the-acquisition (multi-contact manipulations) with 
the aid of a capture interface 1 02. The sensor ean lOl maybe divided, if necessary, into several parts 
in ord e r to accelerate the-capture, with each part being scanned simultaneously. 
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Kindly replace paragraphs [0029] through [0034] with the following: 
When the user removes hi sthe user's fingers from the sensor a the associated cursors are 
destroyed. 

In this manner^ the position and the— development of several fingers are captured 
simultaneously on the sensor. This is a multi-contact capture that is quite innovative for this type of 
controller. 

The sensor us e d for th e e mbodim e nt of the inv e ntion is maybe a resistive tactile matrix tile of 
a known type. 

Resistive tactile matrix tiles are composed of 2 superposed faces on which tracks of ITO 
(indium tin oxide), that is a translucent conductive material, are organized. The tracks are laid out in 
lines on the upper layer and in columns on the lower laye r, formin g and form a matrix (cf. Figur eas 
shown in Fig. 2A}. 

The two conductive layers are insulated from one another by spacing braces. The intersection 
of the line with the column forms a contact point. When a finger is placed on the tile A a column or 
columns situated on the upper layer are put in contact with a line or line situated on the lower layer, 
thus creating one or several contact points (cf. Figur e as shown in Fig. 2B). 

A variant of th e inv e ntion advantag e ously consists in replaoin g lt is possible to replace the 
braces by a transparent resistive material (e.g., a conductive polymer) whose resistance varies as a 
function of the pressure, which resistance drops if a sufficient pressure force is exerted. In this 
manner^ it would is also be-possible to extract the pressure exerted on the surface by performing a 
resistance measurement at each line-column intersection. 
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Kindly replace paragraphs [0036] through [0040] with the following: 

The state of the tile is measured at least 100 times per second, which tile can be divided into 
several zones in order to peFerm perform a parallel processing on these zones. 

Thus, according to the invention the sampling frequency of the tile i smaybe at least 1 00 Hz. 

Another basic element is the electronic device for scanning the tactile tile that allows the 
simultaneous detection of several contact points on the matrix sensor. In fact, the known methods of 
acquisitions for this type of sensor[[s]] do not allow the detection of several simultaneous contact 
points. 

The known methods known in th e past do not allow the problems illustrated by figur e in Fig. 
2C to be solved. 

If a simultaneous measurement of all the lines is performed while feeding a column, 
problems of orthogonality arise. Contact point No. 1 will mask contact point No. 2. Likewise, if a 
line is measured when all the columns are fed, contact point No. 2 is masked by contact point No. 1 . 
The solution propos e d for solving this problem consists is in performing a sequential scanning of the 
sensor. 

Kindly replace paragraph [0042] with the following: 

When one of the columns is placed under voltage^ the others are in high impedance in ord e r 

to prevent the propagation of current into the latter. 

Kindly replace paragraphs [0047] through [0048] with the following: 

As the goal is to form a multi-contact tile, the total scanning of the matrix is carried out at an 

elevated frequency in ord e r to obtain the value of each of the intersection points of the tile several 

times per second. 
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The device permitting the acquisition of the tile data is illustrated in figureFig. 2D, 
representing the algorithm of the acquisition of a tile comprising 100 lines (L) and 135 columns (C). 

Kindly replace paragraphs [0051] through [0061] with the following: 

In figwesFigs. 2E and 2F a the cloud of points absorbs a large part of the electrical potential of 
the fed column. The potential measured at the isolated point is therefore too low to be detected. 

TheA solution to this problem consists is in using a voltage comparator piloted digitally at the 
output of the line in ord e r to determine whether the tension observed is sufficient for being 
considered as resulting from the action of a finger on the tactile tile. The reference value of the 
comparator (comparison threshold) is decremented at each line measure. Thus, the comparison 
values of the last lines are lower than those of the first lines, which allows the contact point located 
at the lower left or the upper right to be detected in the same manner. 

Thus, e.g., thea complete sampling of the tile is performed at least 100 times per second for 
the columns and the lines. 

The data from capture interface 102 thus form an image representative of the totality of the 
sensor. This image is placed in memory so that a program can proceed to the filtering, the detection 
of the fingers and to the creation of the cursors . R e f e r for this e ff e ct to figure as seen in Fig. 1. 

The filtering phase illustrated by SgweFig. 3B consists in e liminating th e eliminates noise 
that might be generated by the acquisition interface or the sensor itself. It is considered that only the 
clouds of several contact points can correspond to the pressure of a finger. Therefore, a bounding 
zone detection is carried out in ord e r to eliminate isolated contact points. 

The following stage consists in assooiatin g associates a cursor with each support point 
(figure Fig. 3C). To this end^ the barycenter of each bounding zone is calculated. When a finger is 
released^ the corresponding cursor is freed. 
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The program executed locally by the main processor allows these cursors to be associated 
with graphical objects that are displayed on screen 105 in order to manipulate them. At the same 
time A the local program uses these cursors for generating control messages addressed to the host 
computer or the controlled apparatus. 

Furthermore, the program comprises a simulator of the physical models allowing the 
modification of the interaction laws between the cursors and the graphical objects. Different 
physical models can be employed: spring-loaded system, vibration of a string, management of 
collisions, the law of gravity, electromagnetic fiel d and the like . 

The program considers the positioning of the cursors and on which graphical object each is 
located. A specific processing is supplied to the data coming from the sensor as a function of the 
object considered. For example, a pressure measurement (corresponding to a development of the 
spot made by the finger on the tactile tile in a short interval of time) can be interpreted. Other 
parameters can be deduced as a function of the nature of the object: the acceleration, speed, 
trajectories, etc. Algorithms of recognition of form can also be applied in order to differentiate 
different fingers. 

The main program 103 also transmits the data to be displayed on screen 105 to graphical 
interface 104. Moreover, this graphical interface is constituted byof a graphical processor. This 
graphical processor is, e.g., of a known type. The latter can be constituted byof primitive graphical 
functions allowing, e.g., the displaying of bitmap, ef-fonts of polygons and e^figures in 2 and 3 
dimensions, the-vectorial design, the-antialiasing, the-texture mapping, the-transparency and the 
interpolation of colors. 

In this pr e sentation of tho invention th eThe main program may also comprise[[s]] an analyzer 
of mathematical expressions that allows mathematical functions to be inputted and calculated in real 
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time. These functions allow the values of any variable to be modified. For example, the coordinates 
(x, y) of a cursor inside an object can be considered as two variables comprised between 0 and 1. 
The expression analyzer allows an expression of the type "x* 1000+600" to be created in order to 
obtain a new variable whose value is comprised between 600 and 1600. The variable obtained 
allows the control, e.g., of the frequency of an oscillator comprised between 600 and 1600 hertz. 
Kindly replace paragraph [0066] with the following: 

It should be noted as illustrated in figur eFig. 1 1 that the Ethernet connection offers the user 
the possibility, by using a simple hub (multi-socket network), of indefinitely expanding histhe 
control apparatus by constituting a network of controllers in accordanc e with th e inv e ntion . 

Kindly replace paragraph [0069] with the following: 

Finally, an interface editor 107 at the level of the computer of the user allows the interface, 
that is, the totality of the graphic objects displayed on screen 105, to be programmed in a graphical 
manner. In this e mbodiment of th e inv e ntion th e The interfaces aremay themselves be organized in 
scenes, that are higher hierarchical structures. In fact, each scene comprises several interfaces. The 
user can interchange the interfaces with the aid of a button keyboard or a control pedal board 
connected to input-output port 109. 

Kindly replace paragraph [0071] with the following: 

The user has at hi sthe user's disposal, e.g., a library of parameterable graphical objects 
allowing the composition of different interfaces according to the application desired. Figufe sFigs. 4 
and 5 represent different graphical objects placed at the disposition of the user. 

Kindly replace paragraph [0074] with the following: 

In addition to the positioning of graphical objects on the main window, other secondary 
windows allow the regulation of different parameters inherent in the objects (graphical properties, 
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physical behavior). For example, a button 402 can also act as a switch or as a trigger. In the case of 
the trigger mode A a pressure measurement can optionally be performed. Another example of a 
parameterable object is area 2D (503, 544) of which the principle consists in cludes moving pawns 
inside a delimited zone. The number of pawns present in area 2D is a parameterable option. The 
area can be configured in uniplan mode, a mode in which the pawns enter into collision with each 
other, or multi-plan, a mode in which the pawns are placed on distinct superposed planes. Physical 
parameters can also be configured: the coefficient of friction of the pawns on the plane, the rebound 
and the attraction of the pawns on the edges and among themselves. 
Kindly replace paragraph [0082] with the following: 

Another variant of area 2D consi s ts in cludes applying a physical law of the "spring-loaded" 
type. A virtual rubber band is stretched between each cursor and each pawn. The user can modify 
the behavior of this object by configuring the friction and the interpolation factor. These properties 
can also be modified in real time with the aid of other objects. 

Kindly replace paragraphs [0084] through [0090] with the following: 

A visualization of different examples of interfaces uniting different types of objects is 
illustrated by figwesFigs. 6 to 9, in which several objects described above can be observed. 

Figure. 6 shows an arrangement of 6 areas 2D (601) containing 1 pawn each. This interface 
could control, e.g., six different filters assigned to one or several sound sources. In this instance^ the 
abscissa movement of each pawn in each zone controls the frequency of the filter whereas the 
ordinate movement controls the quality factor or the width of the filter band. 

Figure. 7 shows an example of the control of a synthesizer or of a sampler of a known type. 
The interface is composed by a tempered keyboard 704 controlling the pitch of the sounds, by a 
group of four vertical potentiometers 703 allowing the control, e.g., of its dynamic envelope (attack 
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time, hold level, release time). An area 2D (701) containing 3 pawns allows the control, e.g., of 
effects applied to the sound (reverberation, echo, filters). A matrix of 16 buttons 792 can, e.g., 
release 16 different recorded musical sequences or also call up 16 prerecorded configurations of the 
previously described controls. 

Another example of an application of th e inv e ntion is illustrated byflgur eFig. 8 showing the 
control of a device for the broadcasting of different sound sources into space on a device constituted 
by several loudspeakers. In this configuration an area 2D (801) representing the broadcasting space 
contains 4 pawns 801 corresponding to four sound sources. Area 2D also contains 5 icons 802 
representing the position of five loudspeakers. The level and/or the phase of each sound source 
relative to each enclosed space is regulated by moving the different pawns 802, which determines its 
emplacement in the space. Moreover, a group of four linear potentiometers 803 allows the relative 
level of each source to be regulated. A unit of four buttons 804 allows each sound source to be 
activated or deactivated. 

Another example is illustrated by figur e in Fig. 9 that shows the control of a synthesizer or a 
sound generator according to a configuration different thanfrom that shown by figur e in Fig. 7. Here, 
the frequency of the sound generator is controlled by four virtual strings 903. The initial tension (the 
pitch) of each string can itself be controlled, e.g., by a linear potentiometer 902. An area 2D 10, e.g., 
control other parameters of the sound generator such as the output level, the sound quality, the 
panning, etc. 

Figure. 1 0 shows the control of equipment for audio and/or video editing of a known type. A 
serrated wheel 1001 allows the rate of reading the audio and/or video sources to be controlled. 
Status display object 1002 allows the positioning of the reading to be represented according to a 
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format (hour, minute, second, image) of a known type. A set of buttons 1003 allows access to the 
functions of reading and editing of the controlled apparatus. 

The inv e ntion was devices and methods described above are b y way of example. It is 
understood that an exp e rt one skilled in the art is capable of realizing different variants of the 
inv e ntion devices and methods without departing from the scope of the paten tappended claims . 
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